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Executive  Summary 


Background:  The  increased  use  of  night  vision  goggles  (NVGs)  by  the  military  fixed  and 
rotary-wing  communities  has  created  an  expanded  need  for  NVG  training.  The  NVG  Training 
Course  is  the  initial  curriculum  for  USAF  and  USN/USMC  NVG  aircrew.  This  training  is 
delivered  using  platform  instruction  in  a  multimedia  classroom,  hands-on  NVG  adjustment 
training  in  an  eye  lane,  and  a  scale  model  terrain  board  and  lighting  system  to  demonstrate 
various  effects  to  students  viewing  with  NVGs.  Advances  in  virtual  environment  technologies 
have  enabled  the  potential  for  computer-based  virtual  training  systems  to  accomplish  the  terrain 
board  learning  objectives.  The  Virtual  Terrain  Board™  (VTB)  takes  advantage  of  advancements 
in  PC-based  graphics  systems  and  high-resolution  projector  capabilities  to  create  a  physics-based 
virtual  NVG  environment  designed  to  allow  the  user  to  accomplish  NVG  training  objectives. 

Introduction:  This  report  describes  the  development  and  evaluation  of  the  Night  Readiness, 

LLC  VTB  (V2.0c)  and  its  instructional  modules  for  NVG  ground  training.  The  evaluation  phase 
of  this  effort  consisted  of  a  training  effectiveness  assessment  of  the  VTB,  based  upon  feedback 
from  NVG  Training  Course  instructors  and  students  currently  using  the  VTB  at  six  military 
installations.  The  primary  objectives  of  the  training  effectiveness  evaluation  were  to  detennine  if 
the  established  terrain  board  learning  objectives  could  be  accomplished  with  the  VTB,  to  identify 
additional  learning  objectives  that  could  be  accomplished  using  the  VTB,  and  to  identify  areas  of 
improvement  required  to  increase  the  training  value  of  the  VTB  system.  The  evaluation  was 
designed  to  elicit  feedback  from  users,  both  qualified  NVG  Training  Course  terrain  board 
instructors  and  students  undergoing  refresher  training  and  students  attending  initial  NVG  ground 
training. 

Methods:  Eleven  currently  qualified  NVG  instructors  from  USN,  USAF,  and  USMC  completed 
both  a  terrain  board  learning  objectives  checklist  and  instructor  survey.  Seventy-six  VTB 
students  completed  a  separate  VTB  student  survey.  Fifty-one  of  the  NVG  students  were  either 
USMC  or  USAF  personnel  completing  their  annual  NVG  refresher  training.  The  remaining  25 
participants  were  students  attending  either  the  NVG  Instructor  Course  or  the  NVG  Training 
Course.  All  student  participants  completed  a  survey  immediately  following  their  VTB  training 
session  conducted  in  NVG  stimulate  mode. 

Results:  All  instructors  reported  the  ability  to  accomplish  the  majority  of  the  terrain  board 
learning  objectives  on  the  checklist  using  the  VTB.  A  few  instructors  indicated  that  they  were 
unable  to  fully  accomplish  specific  learning  objectives  pertaining  to  the  demonstration  of  the 
NVG  effects  of  different  color/wavelengths  of  light,  moon  elevation/angle,  halos,  and  shadows. 
The  overall  usability  of  the  VTB  was  rated  as  effective  by  the  instructors,  although  several 
recommendations  were  provided  to  improve  the  usability  of  the  system.  The  VTB  student 
survey  results  were  consistent  with  the  instructor  results,  and  indicated  that  both  USMC  and 
USAF  refresher  students  rated  the  VTB  as  an  effective  training  system  to  prepare  students  for 
their  assigned  flying  missions. 

Conclusions/Recommendations:  The  instructor  and  student  survey  results  indicate  that  the 
VTB  is  an  effective  training  device  that  can  accomplish  most  aspects  of  the  established  terrain 
board  learning  objectives.  There  were  specific  limitations  cited  by  some  instructors  regarding 
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the  ability  to  fully  demonstrate  the  learning  objectives  pertaining  to  the  different 
color/wavelengths  of  light,  effects  of  different  illumination  levels  on  halos,  and  effects  related  to 
NVG  gain.  It  is  recommended  that  future  VTB  development  efforts  address  these  limitations  in 
the  demonstrations  related  to  NVG  response  to  lighting  effects,  by  exploring  the  use  of  an 
auxiliary  lighting  system  or  by  enhancing  the  capabilities  of  the  VTB.  Additional 
recommendations  aimed  at  improving  system  usability  included  adding  the  capabilities  to 
generate  customized  training  modules  and  a  more  intuitive  ability  and  expanded  capability  to 
change  lunar  phase,  elevation,  and  azimuth  to  aid  in  the  demonstration  of  effects  related  to  moon 
position.  The  addition  of  instructional  modules  to  demonstrate  environmental  and  weather 
effects  and  threat  effects  (e.g.,  ordnance,  lasers)  is  recommended  for  future  VTB  efforts.  The 
development  plans  for  the  VTB  currently  underway  along  with  the  improvements  recommended 
in  this  evaluation  will  provide  a  balanced,  practical  approach  to  improving  the  usability, 
versatility  and  training  effectiveness  of  the  VTB. 
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PREFACE 


This  effort  was  completed  by  the  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate  (AFRL/RHA),  Warfighter  Readiness  Research  Division  with  contractor  support  from 
L-3  Communications  and  The  Boeing  Company.  The  work  was  completed  under  Contract  No. 
FA8650-05-D-6502,  Task  Order  0016.  The  AFRL  contractor  monitor  was  Mr.  Jay  Carroll;  the 
government  principal  investigator  was  Capt  Joelane  E.  Lindberg. 

This  report  documents  the  development  and  evaluation  of  the  Virtual  Terrain  Board™  (VTB) 
(Versions  2.0a-c),  developed  by  Night  Readiness,  LLC.  The  VTB  system  evaluation  was  carried 
out  in  the  June-October  2007  time  period  in  collaboration  with  personnel  from  Marine  Aircraft 
Group  39  (MAG-39),  at  Marine  Corps  Air  Station  (MCAS)  Camp  Pendleton. 

The  information,  results,  conclusions,  and  recommendations  presented  in  this  document  were 
generated  solely  from  the  opinions  of  the  study  participants  and  do  not  represent  USAF  policy  or 
endorsement  of  any  product  or  corporation  mentioned. 

ACKNOWLEDGEMENTS 

Special  acknowledgement  is  due  to  LT  Kimberly  Oelschlager  (USN),  MAG-39,  for  her 
assistance  in  survey  development  and  administration  to  USMC  refresher  students  at  Camp 
Pendleton.  Acknowledgements  are  also  due  to  the  NVG  course  instructors  at  the  participating 
USAF,  USN  and  USMC  locations  who  completed  surveys  and  administered  surveys  to  NVG 
Training  Course  students.  Their  efforts  in  survey  administration  and  diligence  in  returning 
completed  surveys  to  AFRL  are  greatly  appreciated. 

The  authors  also  acknowledge  Capt  Christina  Price  (USAF)  who  provided  valuable  support  to 
this  effort  in  partial  fulfillment  of  her  Master’s  degree  requirements  from  Arizona  State 
University. 


1.  Background 


The  increased  use  of  night  vision  goggles  (NVGs)  by  the  United  States  Air  Force  (USAF),  U.S. 
Navy  (USN),  and  U.S.  Marine  Corps  (USMC)  aviation  communities  has  created  an  expanded 
need  for  NVG  training  (Joralmon,  2004).  The  Warfighter  Readiness  Research  Division  of  the 
Air  Force  Research  Laboratory  (AFRL/RHA),  Mesa  AZ  has  been  developing  and  distributing 
such  training  since  the  early  1990s  (Antonio,  Berkley,  Fiedler,  &  Joralmon,  2004;  Antonio, 
Berkley,  &  Joralmon,  1994,  1998;  Joralmon  &  Antonio,  1992).  One  product,  the  NVG  Training 
Course,  is  the  initial  curriculum  for  USN,  USMC,  and  USAF  aircrew  who  fly  missions  with 
NVGs.  The  NVG  Training  Course  is  not  specific  to  any  one  aircraft  type  as  it  addresses  training 
objectives  common  to  most  tanker,  transport,  helicopter,  fighter,  and  attack  aircraft  (Joralmon, 
2004). 

The  Night  Imaging  and  Threat  Evaluation  Laboratory  (NITE  Lab)  is  a  standardized 
configuration  for  the  physical  layout  of  facilities  used  for  NVG  training  in  the  US  (Joralmon, 
2004).  A  typical  NITE  Lab  consists  of  a  multimedia  classroom,  an  eye  lane  where  NVG 
adjustment  procedures  are  taught,  and  a  terrain  board  used  to  demonstrate  various  NVG  effects. 
The  terrain  board  is  a  large  (10’  by  10’)  model  layout  of  various  types  of  terrain  (e.g.,  desert, 
mountains,  and  oceans).  Within  this  varied  terrain  are  different  cultural  features  such  as  cities, 
farms,  power  lines,  airports,  ships,  and  bridges.  A  lighting  system  over  the  terrain  board  enables 
the  instructor  to  vary  the  night  illumination  level  between  overcast  starlight  and  full  moon.  A 
photograph  of  an  instructor  using  a  typical  terrain  board  is  shown  in  Figure  1 .  The  instructor 
demonstrates  to  students,  who  gather  around  the  board  and  view  with  NVGs,  various  NVG 
effects  such  as  shadowing,  cultural  lighting,  and  terrain  albedo. 


Figure  1.  Photograph  of  a  NVG  Training  Course  instructor  and  students  using  a  terrain  board 
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Advances  in  visual  display  technologies  and  virtual  environment  interfaces  have  enabled  the 
potential  for  computer-based  virtual  training  systems  to  accomplish  NVG  training  curriculum. 
The  Virtual  Terrain  Board™  (VTB)  developed  by  Night  Readiness,  LLC  takes  advantage  of 
advancements  in  personal  computer  (PC)-based  graphics  systems  and  high-resolution  projector 
capabilities  to  create  a  physics-based  virtual  rendering  of  the  NVG  environment.  In  the  early 
development  stages  of  the  VTB,  high-resolution,  daytime  digital  images  of  real-world  locations 
were  combined  to  create  a  360-degree  panoramic  photo  document  of  the  area.  These  photos 
were  used  as  a  basis  for  rendering  a  nighttime  scene.  Material  response  data  was  processed  to 
provide  albedo  and  reflectance  for  each  texel  in  the  scene.  Additional  features,  such  as  night 
sky,  horizon  glow,  halos,  gain,  and  noise  were  combined  with  the  images  to  create  NVG  unique 
effects.  Further  development  of  the  VTB  brought  about  more  location  databases  (e.g.,  Whiting 
Field,  Camp  Pendleton)  created  by  using  the  same  type  of  photo  manipulation  and  rendering  to 
illustrate  different  terrain  types,  illuminations,  contrasts  and  other  NVG  affecting  variables  in 
various  scenes. 

The  scenes  produced  by  the  VTB  provide  untethered  NVG  simulation  in  two  modes:  simulate 
and  stimulate.  The  simulate  mode  displays  a  NVG  scene  that  can  be  viewed  with  the  naked  eye. 
The  NVG  unaided  audience  may  view  the  scene  displayed  on  the  projector  screen.  In  the 
stimulate  mode,  the  projected  image  stimulates  the  NVG  to  present  a  real-world  NVG-aided 
scene  to  the  user.  The  user  can  also  see  a  rendering  of  a  realistic  unaided  (out-the-window) 
nighttime  scene  when  looking  under  or  around  the  goggles.  An  initial  VTB  system  was  installed 
in  the  Night  Operations  Center  of  Excellence  (NOCOE)  research  facility  at  AFRL/RHA  in 
January  2007.  This  initial  version  of  the  VTB  leveraged  the  results  of  research  and  engineering 
development  efforts  of  the  AFRL  NVG  training  research  team  and  subcontractor  Renaissance 
Sciences  Corporation  (RSC)  accomplished  in  the  2004-2005  timeframe.  A  typical  configuration 
of  the  VTB  in  a  NVG  training  classroom  environment  is  depicted  in  Figure  2.  An  example  of  a 
nighttime  NVG-aided  scene  in  the  simulate  mode  of  the  VTB  is  provided  in  Figure  3. 


Figure  2.  Typical  configuration  of  the  VTB  system  in  a  NVG  training  classroom  environment 


11 


Figure  3.  VTB  scene  of  power  lines  in  mountainous  terrain  in  NVG  simulation  mode 

(from  Night  Readiness,  2007) 


2.  Scope 

This  report  describes  the  development  and  evaluation  of  the  VTB  (versions  2.0a  thru  2.0c)  and 
its  instructional  modules  for  NVG  ground  training.  The  VTB  development  efforts  encompassed 
the  phases  of  measurement/calibration  of  VTB  display  system  settings  for  NVG  stimulation  and 
simulation,  development  of  advanced  curriculum  modules  to  accomplish  the  training  objectives 
of  the  physical  terrain  board,  delivery  of  a  simulation  mode  and  providing  technical  support  to 
the  collection  and  delivery  of  NVG  video  and  digital  imagery  for  use  in  continued  database 
development  and  evaluation. 

The  evaluation  phase  of  this  effort  consisted  of  a  training  effectiveness  assessment  of  the  VTB, 
based  upon  user  feedback  from  NVG  Training  Course  instructors  and  students  currently  using 
the  VTB  at  six  USAF,  USN,  and  USMC  installations.  The  training  effectiveness  evaluation 
assessed  the  effectiveness  of  the  VTB  (V2.0c)  training  modules  and  instructor  guide  in 
accomplishing  the  terrain  board  learning  objectives  established  for  the  NVG  Training  Course 
syllabus.  These  objectives  include  the  demonstration  of  the  effects  of  illumination  and  contrast 
over  different  terrains,  moon  elevation  and  angle,  shadows,  and  cultural/aircraft  lighting  on  NVG 
operations.  Recommendations  for  future  versions  of  the  VTB  relevant  to  design  considerations, 
database  features,  and  instructional  modules  were  also  addressed  in  the  evaluation. 

3.  System  Description 

The  major  components  of  the  VTB  system  are  a  3-chip  Panasonic  digital  light  processing  (DLP) 
projector,  driven  by  a  Dell  Dimension  9200  desktop  computer.  A  wireless  Dell  keyboard  and 
mouse  are  used  for  basic  computer  entry  and  to  access  the  VTB  runtime  software.  A  Logitech 
Cordless  Rumblepad  2  (Model  No.  C-UE10)  gamepad  is  used  for  menu  control  and  scene-to- 
scene  navigation.  This  controller  also  allows  the  user  to  zoom,  pan,  and  turn  entities  on/off 
within  a  scene.  A  custom  filter  is  fitted  over  the  projector  lens  to  reduce  aberrant  color  contrast 
effects  and  optimize  the  black  level  of  the  projector.  The  projected  scene  is  presented  on  an 
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opaque  black  fabric  screen  (approximately  8  ft  wide  x  5  ft  high)  mounted  on  a  wall  12  or  15  ft 
(depending  upon  location)  in  front  of  the  projector.  A  MenuCube™  software  program  serves  as 
the  interface  for  instructors  to  browse  through  the  various  database  scenes  and  training  modules. 

As  described  in  the  VTB  Instructor  Guide  (May  2007),  the  runtime  software  (V2.0)  processes  the 
database  for  display.  Each  of  the  three  DLP  mirrors  in  the  projector  projects  a  different  portion 
of  the  spectrum  on  the  screen.  One  channel  displays  emissive  objects  such  as  cultural  lights, 
moon,  and  stars.  A  second  channel  displays  reflective  surfaces  such  as  terrain  and  specific 
objects,  and  the  third  channel  displays  noise. 

The  three  VTB  training  modules  implemented  at  the  time  of  this  evaluation  were  entitled 
Illumination  and  Contrast,  Shadows,  and  Lighting  Effects.  Each  training  module  is  comprised  of 
several  database  scenes  that  are  selected  to  highlight  specific  training  objectives  and  teaching 
points.  Each  scene  is  presented  in  a  slideshow  presentation  guided  by  the  instructor.  A  graphic 
rendering  of  the  basic  VTB  system  architecture  provided  by  Night  Readiness,  LLC  is  displayed 
in  Figure  4. 


Instructor  Keyboard 


Instructor  Controller 


Student  NVGs 


Figure  4.  VTB  system  architecture  (from  Night  Readiness,  2007). 


4.  VTB  Instructional  Module  Development 

The  initial  version  of  the  VTB  training  modules  software  (V2.0a)  and  supporting  instructional 
materials  were  delivered  to  AFRL  in  April  2007.  The  training  modules  delivered  with  this  initial 
version  of  the  VTB  addressed  the  legacy  learning  objectives  established  for  the  NITE  lab 
physical  terrain  boards.  The  modules  of  illumination  and  contrast,  lighting  effects,  and  shadows 
contained  specific  scenes  and  associated  teaching  points  aimed  at  each  of  the  terrain  board 
objectives.  The  illumination  and  contrast  module  was  designed  to  demonstrate  the  effects  of 
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various  natural  illumination  levels  (from  full  moon  to  starlight)  over  various  types  of  terrain. 
Specific  objectives  included  the  demonstration  of  illumination  level  on  NVG  image  quality, 
differences  in  albedo  for  various  objects,  and  the  appearance  of  power  lines  at  different 
illumination  levels.  The  objective  of  the  lighting  effects  module  was  to  demonstrate  how 
different  light  sources  affect  the  quality  of  the  image  as  well  as  objects  within  the  scenes. 

Specific  objectives  included  the  demonstration  of  the  effects  of  different  color/wavelengths  of 
light  on  the  NVG  image,  estimating  distances  to  point  light,  demonstration  of  cultural  lighting 
effects  on  NVGs.  The  objective  of  the  shadows  module  is  to  demonstrate  the  effects  of  moon 
elevation  angle  and  position  on  shadowing  of  objects  in  the  NVG  image. 

A  variety  of  geographical  locations  were  used  in  the  database  scene  development  for  each 
training  module.  These  locations  included  a  series  of  navigation  points  along  the  “Green  Route” 
near  Naval  Air  Station  (NAS)  Whiting  Field,  mountainous  terrain  scenes  from  Usery  Mountain 
Pass  and  Four  Horse  Mesa  in  Arizona,  and  scenes  derived  from  satellite  imagery  from  the 
Mercury,  NV  region  near  Nellis  AFB.  The  AFRL  NVG  research  team  provided  feedback  to 
Night  Readiness  LLC  with  respect  to  content  and  format  of  the  VTB  Instructor  Guide  as  well  as 
desired  upgrades  to  the  database  of  scenes  established  for  each  of  the  training  modules  described 
above. 

During  the  quarterly  period  ending  30  June  2007,  Night  Readiness  upgraded  the  VTB  software 
to  version  2.0c.  This  version  included  a  simulate  mode  and  updates  to  the  training  modules  of 
illumination  and  contrast,  shadows,  and  lighting  effects.  The  incorporation  of  the  simulate  mode 
allows  for  a  standard  classroom  or  desktop  delivery  of  ah  existing  VTB  content.  The  simulate 
mode,  intended  for  unaided  (non-NVG)  viewing,  renders  ah  database  scenes  as  they  would 
appear  to  a  user  viewing  with  NVGs.  A  revised  VTB  Instructor  Guide  outlining  training 
objectives  and  teaching  methods  for  the  first  VTB  instructional  modules  was  delivered  during 
this  period. 

NVG  research  team  personnel  provided  additional  technical  support  in  defining  requirements  for 
a  database  of  NVG  scenes  tailored  to  the  Marine  Corp  Air  Station  (MCAS)  Camp  Pendleton  area 
to  support  the  specific  training  needs  of  that  VTB  installation.  In  September  2007,  a  NVG 
research  team  representative  participated  with  Night  Readiness  LLC  in  the  data  collection  of 
high  resolution  digital  imagery  using  a  remote  helicopter  equipped  with  photographic  equipment. 
The  digital  photographs  of  specific  MCAS  Camp  Pendleton  navigation  points  (i.e.,  intersections, 
beach,  lake,  and  407  Firing  Range)  were  processed  to  develop  nighttime  scenes  at  various 
illumination  levels  and  added  to  the  VTB  scene  database. 


5.  VTB  Training  Effectiveness  Evaluation 
5.1  Introduction 

The  terrain  board  has  been  an  integral  part  of  NVG  ground  training  for  the  USN,  USMC  and 
USAF  since  its  inception.  The  primary  learning  objectives  of  the  terrain  board  instructional 
module  include  the  demonstration  of  various  NVG  effects.  The  primary  terrain  board  learning 
objectives  are  listed  below: 
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•  Effects  of  various  illumination  levels  on  contrast/detail  over  different  terrains 

•  Effects  of  different  color/wavelengths  of  light  under  different  illumination 
levels  on  the  NVG  image 

•  Albedos  of  different  objects  in  high  and  low  contrast  areas 

•  Appearance  of  power  lines 

•  The  effects  of  different  illumination  levels  on  halos 

•  Effects  of  shadows  on  the  NVG  image 

•  Effects  of  moon  elevation  and  angle  on  the  NVG  image 

•  Differences  between  high  and  low  contrast  terrain 

•  Cultural  features  such  as  roads,  power  lines,  runways,  and  boats 

•  Effects  of  cultural  lighting 

Each  objective  is  addressed  using  a  variety  of  teaching/demonstration  points,  depending  upon 
instructor  preference  and  user-specific  requirements.  In  a  survey  conducted  by  AFRL  to  define 
NVG  training  requirements  for  undergraduate  pilot  training,  516  Instructor  Pilots  (IPs)  with 
current  or  previous  NVG  qualification  rank  ordered  the  three  learning  tools  that  were  most  useful 
to  them  during  their  NVG  training  (Martin,  Berkley,  Riegler,  Good,  Anderson,  Torruella,  et  al, 
2004).  The  terrain  board  was  ranked  third  by  the  IPs,  behind  flight  training  and 
platform/simulation  training.  This  indicates  that  NVG-experienced,  qualified  pilots  consider  the 
terrain  board  a  valuable  NVG  ground  training  tool. 

A  major  conclusion  from  the  Martin  et  al,  (2004)  survey  was  that  realistic  NVG  simulation  has  a 
“unique  potential  for  both  highly  effective  and  relatively  inexpensive  training.”  These  authors 
recommended  that  undergraduate  pilot  training  should  include  NVG  simulation  exposure.  The 
development  and  evaluation  of  the  VTB  system  described  in  this  effort  was  a  significant  step  in 
the  realization  of  that  goal. 

The  primary  objectives  of  the  training  effectiveness  evaluation  were  to  determine  if  the 
established  terrain  board  learning  objectives  could  be  accomplished  with  the  VTB,  to  identify 
additional  learning  objectives  that  could  be  accomplished  using  the  VTB,  and  to  identify  areas  of 
improvement  required  to  increase  the  training  value  of  the  VTB  system.  The  evaluation  was 
designed  to  elicit  feedback  from  users,  both  qualified  NVG  Training  Course  terrain  board 
instructors  and  students  undergoing  refresher  training  and  students  attending  initial  NVG  ground 
training. 

Instructor  and  student  surveys  were  developed  by  the  AFRL  NVG  research  team  to  obtain  this 
information  from  VTB  instructors  and  students  at  six  USMC,  USN,  and  USAF  installations.  The 
evaluation  was  conducted  in  collaboration  with  Marine  Aircraft  Group  39  (MAG-39)  at  Camp 
Pendleton,  CA  who  provided  feedback  from  several  NVG  Refresher  Course  students  who 
received  VTB  demonstrations/training  during  their  annual  refresher  training.  These  USMC 
refresher  students  consisted  of  rotary-wing  pilots  (UH-1N,  CH-46E,  and  AH-1W).  C-17  aircrew 
refresher  students  at  Travis  Air  Force  Base  (AFB)  also  participated  in  the  evaluation.  NVG 
Instructor  Course  students  from  Randolph  AFB  and  Travis  AFB  also  participated. 
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5.2  Method 

5.2.1  Participants 

Eleven  currently  qualified  NVG  Training  Course  instructors  from  USN,  USAF,  and  USMC 
participated  in  the  evaluation.  All  instructors  had  prior  teaching  experience  using  the  physical 
terrain  board  and  all  had  hands-on  teaching  experience  with  the  VTB  (V2.0c)  at  their  NVG 
training  site.  Seventy-six  VTB  students  participated  in  the  evaluation  at  the  various  USN, 
USMC,  and  USAF  training  sites.  Fifty-one  of  the  NVG  students  were  either  USMC  or  USAF 
personnel  completing  their  annual  NVG  refresher  training.  The  remaining  25  participants  were 
students  attending  the  NVG  Training  Course  or  the  NVG  Instructor  Course. 

5.2.2  Surveys 

The  instructor  survey  (Appendix  A)  addressed  VTB-user  interface  issues,  ratings  of  training 
effectiveness  in  accomplishing  terrain  board  learning  objectives,  learning  objectives  that  were 
not  possible  to  accomplish  on  the  VTB  that  can  be  demonstrated  on  the  physical  terrain  board 
and  vice  versa,  and  recommended  improvements  to  VTB  training  modules  and  system 
capabilities.  The  terrain  board  learning  objectives  checklist  (Appendix  B)  was  a  one-page 
checklist  of  specific  terrain  board  learning  objectives  designed  to  elicit  a  “Yes”  or  “No”  response 
regarding  whether  or  not  the  VTB  allowed  them  to  accomplish  each  specific  objective.  The 
VTB  student  survey  (Appendix  C)  addressed  the  effectiveness  of  the  VTB  in  familiarizing 
students  with  each  learning  objective,  compared  the  VTB  to  the  physical  terrain  board  in  ability 
to  accomplish  learning  objectives,  recommended  improvements,  and  the  effectiveness  of  the 
VTB  in  familiarizing  a  student  with  their  assigned  flying  mission. 

5.2.3  Procedure 

The  VTB  training  effectiveness  evaluation  was  officially  “kicked  off’  in  June  2007.  A  request 
for  participation  and  instructor  and  student  surveys  was  sent  by  the  AFRF  NVG  research  team  to 
the  following  NVG  Training  Course  sites: 

•  Marine  Aircraft  Group  39,  MCAS  Camp  Pendleton 

•  Travis  AFB  Tactics,  60  OSS/OSK,  Travis  AFB,  CA 

•  Command  Training  Air  Wing  Five  (CTW-5)  NAS  Whiting,  Field,  FF 

•  56th  Training  Squadron  (TRS),  Fuke  AFB,  AZ 

•  AETC  NVG  Instructor  Course  Schoolhouse  (12  ADS/SGGT)  Randolph  AFB,  TX 

•  AFRF/RHA,  Mesa  AZ 

The  support  requested  from  each  NVG  Training  Course  instructor  POC  is  outlined  below: 

•  Administration  of  the  instructor  survey  and  terrain  board  learning  objectives 
checklist  to  NVG  course  instructors  with  terrain  board  instructional  experience 
with  both  the  virtual  and  physical  terrain  boards. 

•  Administration  of  the  student  survey  to  NVG  course  students  immediately 
following  their  demonstration  of  the  VTB. 

•  Return  completed  instructor  and  student  surveys 
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•  Availability  for  follow-up  interviews  with  instructors  if  necessary 

The  VTB  was  demonstrated  to  students  receiving  initial  and  refresher  NVG  training  in  the 
stimulate  mode.  The  demonstrations  and  surveys  were  administered  by  the  instructors  at  the 
various  NVG  training  sites.  Completed  surveys  were  returned  via  US  mail  or  e-mail  to  AFRL 
during  the  September  to  December  2007  timeframe. 

5.3  Results 

A  summary  of  the  instructor  and  student  survey  respondents  from  each  VTB  training  location  is 
provided  in  Table  1. 

Table  1.  Completed  instructor  and  student  surveys  received  from  each  VTB  training  location 

participating  in  the  evaluation 


Location 

VTB  Instructors 

VTB  Students 

Refresher 

Initial 

NAS  Whiting  Field 

5 

- 

- 

MCAS  Camp  Pendleton 

1 

31 

- 

Luke  AFB 

2 

- 

- 

Travis  AFB 

1 

20 

18 

AFRL,  Mesa 

1 

Randolph  AFB 

1 

- 

7 

Totals 

11 

51 

25 

5.3.1  VTB  Instructor  Survey  Results 

Eleven  instructors  from  various  services  including  the  USAF,  USN  and  USMC  completed  the 
VTB  instructor  survey.  These  individuals  represented  the  TH-57,  C-17,  F-16,  C-5,  AH-1  and 
CH-46  aircraft  communities. 

All  of  the  instructors  that  completed  the  survey  are  currently  qualified  as  terrain  board 
instructors.  A  total  of  5  of  1 1  of  those  instructors  responded  that  their  physical  terrain  board 
experience  exceeded  100  hours.  Six  of  those  surveyed  had  8  hours  or  more  of  VTB  experience. 

Terrain  Board  Learning  Objectives  Checklist 

Ten  of  the  eleven  instructors  who  participated  in  the  evaluation  completed  a  terrain  board 
learning  objectives  checklist.  Instructors  used  the  checklist  to  indicate  “Yes”  or  “No”  as  to 
whether  the  VTB  allowed  them  to  accomplish  specific  learning  objectives.  Instructors  also 
provided  comments  for  the  learning  objectives  receiving  a  “No”  response.  The  results  of  the 
terrain  board  learning  objectives  checklist  are  provided  in  the  Table  2. 
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Table  2.  Terrain  board  learning  objectives  checklist  results 


Terrain  Board  Learning  Objectives 

Indicate  whether  the  VTB  allows  you  to  accomplish  each  specific 
learning  objective. 

No.  of 
Instructors 
responding 
YES 

No.  of 
Instructors 
responding 
NO 

1.  Demonstrate  differences  in  albedo  for  different  objects 

10 

0 

2.  Demonstrate  the  appearance  of  power  lines 

9 

0 

3.  Demonstrate  effects  of  low  illumination  on  contrast/detail 
over  different  terrains 

10 

0 

4.  Demonstrate  the  effect  of  no  illumination 

8 

0 

5.  Demonstrate  effects  of  different  color/wavelengths  of  light  on 
the  NVG  image 

9 

1 

6.  Demonstrate  effects  of  different  color/wavelengths  of  light 
under  different  illumination  levels  on  the  NVG  image 

6 

3 

7.  Demonstrate  effects  of  different  illumination  levels  on  halos 

7 

3 

8.  Demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG 
image 

7 

3 

9.  Demonstrate  differences  of  albedos  in  high  and  low  contrast 
areas 

10 

0 

10.  Demonstrate  effects  of  shadows  on  the  NVG  image 

7 

3 

1 1 .  Demonstrate  the  effects  of  various  illumination  levels  on 
image  quality 

10 

0 

As  evident  in  Table  2,  all  instructors  reported  that  they  were  able  to  accomplish  the  majority  of 
the  terrain  board  learning  objectives  using  the  VTB.  The  terrain  board  objectives  receiving  a 
“No”  response  by  one  to  three  instructors  were  the  VTB’s  ability  to;  a)  demonstrate  effects  of 
different  color  wavelengths,  b)  demonstrate  effects  of  different  color/wavelengths  of  light  under 
different  illumination  levels  on  the  NVG  image,  c)  demonstrate  effects  of  different  illumination 
levels  on  halos,  d)  demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG  image  and  e) 
demonstrate  effects  of  shadows  on  the  NVG  image. 

Demonstration  of  the  effect  of  no  illumination  was  not  rated  by  2  instructors.  They  did  provide 
comments  that  a  scene  presenting  the  effect  of  no  illumination  is  not  available  in  the  VTB 
modules.  One  instructor  suggested  that  this  can  be  demonstrated  by  powering  the  NVGs  on  in  a 
dark,  light-tight  room. 

The  comments  provided  by  the  instructors  who  rated  low  regarding  effects  of  different  color 
wavelengths  (objectives  5  &  6)  indicated  that  the  effect  of  the  lights  needed  to  be  better 
represented  for  the  unaided  view  and/or  in  the  NVG  image.  For  objective  5,  a  suggestion  was 
made  that  incompatible  lights  should  be  shown  in  a  darker  shade  of  red  so  as  to  be  more  easily 
noticed  unaided.  One  instructor  noted  that  halo  size  and  brightness  of  non-compatible  lights 
needs  to  be  included  in  the  VTB.  Another  commented  that  there  was  never  a  difference  in  halo 
effects  or  any  changes  in  the  NVG  image  between  the  various  scenes.  One  instructor  offered  the 
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suggestion  that  on  the  fly  illumination  control  would  alleviate  the  problem  that  he  noted  for 
objective  6. 

The  demonstration  of  the  effects  of  different  illumination  levels  on  halos  was  rated  ineffective  by 

3  instructors.  One  instructor  noted  that  the  halos  at  different  illumination  levels  were  not 
accurate  for  the  illumination  level  (e.g.  opaque  halos  at  high  illumination,  not  opaque  enough  at 
low  illumination).  The  same  instructor  stated  that  when  zooming  in  on  a  light,  halo  size  actually 
decreases  in  the  demonstration.  A  third  instructor  stated  that  the  effect  of  different  illumination 
levels  is  not  "dramatic"  enough  but  should  work  better  if  weather  effects  are  added  to  the  VTB's 
capabilities. 

Three  instructors  rated  the  effect  of  moon  elevation  and  angle  demonstration  on  the  NVG  image 
as  ineffective.  Items  that  arose  for  comment  from  instructors  included:  goggles  do  not  gain 
down  when  directly  viewing  the  moon,  and  that  the  capability  to  change  moon  elevation  while 
looking  at  the  scene  is  desired. 

Three  instructors  reported  that  the  VTB  did  not  allow  them  to  fully  accomplish  the 
demonstration  of  shadows  on  the  NVG  image.  Their  comments  included  the  following:  there  are 
not  enough  modules  with  shadows  and  effects  of  objects  in  shadows,  the  moon  needs  to  be  in 
field-of-view  (FOV)  and  some  shadows  do  not  match  with  other  shadows  in  scene  and  capability 
to  change  moon  while  viewing  the  scene  is  needed. 

Instructors  Survey 

Usability  of  VTB: 

Instructors  rated  the  ease  of  use  of  the  overall  VTB  system  using  a  5 -point  scale  ranging  from 
very  difficult  (1)  to  very  easy  (5).  The  instructors  gave  an  average  score  of  3.8  for  the  overall 
ease  of  use  of  the  VTB  system  from  the  instructor's  point  of  view. 

The  instructors  also  provided  suggested  improvements  to  the  VTB  regarding  its  usability  with 
respect  to  the  video  controller,  menu  navigation,  classroom  environment  and  the  teaching 
modules  provided.  Three  instructors  commented  that  the  video  controller  would  be  improved  by 
controller  modifications  for  one  handed/hands  free  control  and  increased  navigation  speed.  For 
menu  navigation,  8  instructors  suggested  the  ability  to  edit/customize  the  modules  and  menu  and 

4  instructors  noted  that  an  increase  in  scene  to  scene  transition  speed  would  help.  The  classroom 
environment  section  included  instructor's  preferences  for  stadium  seating  configuration  and  the 
ceiling  mounted  projector  for  an  unobstructed  field  of  view  (FOV)  for  each  student.  Five 
instructors  commented  that  the  usability  of  the  teaching  modules  provided  could  be  improved  by 
customization  with  respect  to  the  order  of  scenes  or  the  use  of  site  specific  databases.  Additional 
teaching  modules  recommended  were  addition  of  medium  altitude  shot,  littoral  zone  (ocean/sea), 
weather,  threat  and  time  lapse  modules  by  1  instructor  each. 
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Training  Effectiveness  Ratings: 

Instructors  also  the  rated  the  training  effectiveness  of  twelve  terrain  board  learning  objective 
using  the  5-point  rating  scale  below: 

1.  Very  ineffective  -  unable  to  accomplish  any  learning  objectives 

2.  Ineffective-  unable  to  accomplish  most  learning  objectives 

3.  Adequate  -  able  to  accomplish  some  learning  objectives 

4.  Effective  -  able  to  accomplish  most  learning  objectives 

5.  Very  effective  -  able  to  accomplish  all  learning  objectives 

Although  it  is  not  a  defined  terrain  board  learning  objective,  the  effects  of  environmental  factors 
was  included  in  this  section  to  obtain  instructor  feedback.  Environmental  effects  are  considered 
additional  teaching  points  and  a  feature  that  the  military  services  would  like  to  incorporate  into 
future  terrain  board  modules.  Environmental  effects  are  not  currently  a  function  of  the  VTB 
(V2.0c)  or  the  physical  terrain  board,  although  a  limited  weather  simulation  was  available  on 
some  physical  terrain  boards  at  various  NITE  labs.  This  simulation  consisted  of  a  “window”  that 
the  NVG  user  viewed  the  terrain  board  through  to  simulate  different  weather  effects  such  as  fog 
and  clouds.  None  of  these  weather  windows  are  currently  being  used  and  they  are  no  longer 
available. 

The  surveys  yielded  ratings  that  averaged  1.5  (between  ineffective  and  very  ineffective)  for  the 
effects  of  environmental  factors.  Many  comments  regarding  the  desire  for  environmental  effects 
as  a  capability  of  the  VTB  were  prompted  by  the  inclusion  of  the  feature.  The  majority  of 
instructors  gave  very  ineffective  ratings  in  this  area  due  to  the  complete  absence  of 
environmental  effects  from  the  VTB.  Seven  instructors  noted  either  the  lack  of  environmental 
effects  or  that  environmental  effects  would  be  a  beneficial  feature  to  the  VTB.  The  remaining 
items  in  the  Table  3  are  currently  established  learning  objectives. 

The  learning  objectives  rated  by  the  instructors  were  similar  to  the  objectives  cited  in  Table  2 
above,  with  the  addition  of  cultural  lighting  and  cultural  features.  The  average  ratings  (of  1 1 
VTB  instructors)  for  each  terrain  board  objective  are  provided  in  Table  3.  All  of  the  learning 
objectives  listed  (excludes  environmental  effects)  had  an  average  rating  above  3.5.  The  majority 
of  them  were  rated  above  4.0,  an  effective  rating. 
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Table  3.  VTB  instructors  average  training  effectiveness  rating  for  each  terrain  board  learning 

objective 


Terrain  Board  Learning  Objectives 

VTB  (V2.0c) 

Rate  the  effectiveness  of  the  VTB  in  accomplishing  the  terrain  board 

learning  objectives: 

Instructors’ 
Average 
Rating  of 
Training 
Effectiveness 

1.  Demonstrate  both  high  and  low  contrast  terrain 

4.5 

2.  Demonstrate  the  appearance  of  power  lines 

4.0 

3.  Demonstrate  effects  of  illumination  level  on  contrast/detail  over 
different  terrains 

4.4 

4.  Demonstrate  the  effect  of  no  illumination 

4.1 

5.  Demonstrate  effects  of  different  color/wavelengths  of  light  on  the 
NVG  image 

4.0 

6.  Demonstrate  effects  of  different  color/wavelengths  of  light  under 
different  illumination  levels  on  the  NVG  image 

3.7 

7.  Demonstrate  effects  of  different  illumination  levels  on  halos 

4.1 

8.  Demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG  image 

3.8 

9.  Demonstrate  differences  of  albedos  in  high  and  low  contrast  areas 

4.3 

10.  Demonstrate  effects  of  shadows  on  the  NVG  image 

4.0 

11.  Demonstrate  the  appearance  of  cultural  features 

4.5 

12.  Culturing  lighting  effects 

3.9 

13.  Demonstrate  effects  of  environmental  factors  (weather,  dust,  smoke) 

1.5 

The  instructors  rated  the  extent  to  which  the  VTB  allowed  them  to  accomplish  the  overall 
learning  objectives  as  compared  to  the  physical  terrain  board.  The  average  rating  for  the  1 1 
instructors  was  4.5. 

The  instructors  were  then  asked  to  explain  the  limitations  of  those  learning  objectives  marked 
ineffective  or  very  ineffective.  Only  one  instructor  rated  any  learning  objectives  (other  than 
environmental  factors)  as  ineffective.  This  instructor  rated  both  the  demonstration  of  effects  of 
different  color/wavelengths  of  light  on  the  NVG  image  and  this  same  demonstration  at  different 
illumination  levels  as  ineffective. 

One  instructor  elaborated  on  limitations  of  those  items  that  he  rated  as  3,  adequate.  He  rated  the 
following  as  adequate:  demonstrate  effects  of  different  color/wavelengths  of  light  on  the  NVG 
image,  demonstrate  effects  of  different  color/wavelengths  of  light  under  different  illumination 
levels  on  an  NVG  image,  demonstrate  effects  of  different  illumination  levels  on  halos  and 
demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG  image.  He  addressed  these  by 
stating  that  for  the  VTB,  colored  lights  are  difficult  to  detect  unaided,  goggle  gain  down  is  not 
demonstrated  well  when  viewing  an  incompatible  light  source,  cultural  lighting  does  not 
illuminate  surrounding  areas  adequately,  there  are  power  line  visibility  issues,  halo  size 
decreases  when  moving  towards  a  light,  lights’  effects  on  halos  are  too  opaque  or  too  bright,  and 
overcast  starlight  is  not  demonstrated. 
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Instructors  listed  any  additional  NVG  training  objectives  that  could  be  effectively  trained  using 
the  VTB.  Two  instructors  mentioned  mission  rehearsal/execution  enhancement  as  another 
potential  learning  objective.  Additional  learning  objectives  noted  by  instructors  were  fly-through 
capability  to  improve  SA  and  motion  cues,  NVG  limitations,  threat  demonstrations,  demonstrate 
aircraft  formation  lights,  environmental  factors  and  motion  illusion  with  moving/stationary  light 
sources. 

Instructors  were  asked  to  list  any  learning  objectives  that  are  not  possible  to  accomplish  on  the 
physical  terrain  board  that  can  be  demonstrated  on  the  VTB.  The  instructors  did  not  identify  any 
specific  learning  objectives  but  instead  provided  several  features/capabilities  of  the  VTB  not 
possible  on  the  physical  terrain  board.  These  are  listed  below: 

•  Cultural  lighting  effects  on  goggle  gain 

•  Zoom  and  360  degree  scan  capability;  high  fidelity 

•  Capability  for  mission  preview 

•  Capability  for  weather,  rain,  dust,  smoke 

•  Illusions  pertaining  to  distance  estimation  to  light  sources 

•  Perspective;  ability  to  have  a  large  class  see  terrain  board  from  the  same  angle 

•  Increased  number/types  of  database  environments 

•  Water  reflection 

•  AAA  fire/gunfire 

•  Moving  vehicles 

Instructors  were  asked  to  list  any  learning  objectives  that  are  not  possible  to  accomplish  on  the 
VTB  that  can  be  demonstrated  on  the  physical  terrain  board.  No  specific  learning  objectives 
were  identified,  but  the  instructors  again  provided  feedback  with  respect  to  features/capabilities 
of  the  physical  terrain  board  not  possible  on  the  VTB: 

•  Depth  perception 

•  Effects  related  to  moon  angle/position 

o  Time  lapsed  moon  sequence  to  demonstrate  effect  of  shadows 
o  Better  demonstration  of  different  angles  of  the  moon  and  shadows 
o  Ability  to  easily  change  moon  angle 
o  Moon  position;  corresponding  shadows 
o  On  the  fly  moon  azimuth  change  and  perspective  shift 
o  Moon  azimuth’s  effect  upon  visibility  of  power  lines 

•  Aircraft/ship  lighting 

•  VTB  is  limited  by  pre-loaded  modules;  Physical  terrain  board  has  flexibility  during 
presentation 

•  Mission  rehearsal 

•  Goggle  gain  effects  while  viewing  incompatible  light  source 

•  Halo  size  change  when  moving  toward  object 


22 


Six  instructors  noted  that  on  the  fly  changes  were  possible  on  the  terrain  board  but  not  the  VTB. 
This  included  changes  in  moon  angle,  azimuth  and  perspective.  Other  responses  varied  greatly 
as  can  be  seen  in  the  list  above. 

The  instructors  then  compared  the  physical  terrain  board  to  the  VTB  with  respect  to  each 
system’s  effectiveness  as  a  stand-alone  device  or  if  their  combined  use  would  provide  the  most 
effective  training.  A  total  of  1 1  instructors  provided  a  response  for  this  question.  The  total 
statements  chosen  by  all  instructors  was  higher  than  1 1  because  the  instructor  was  allowed  to 
select  more  than  one  statement  for  this  question. 

Seven  instructors  chose  A,  the  VTB  is  effective  as  a  stand  alone  training  device.  Five  instructors 
chose  E,  the  virtual  and  physical  terrain  boards  together  provide  the  most  effective  training. 

Four  instructors  chose  C,  the  physical  terrain  board  is  effective  as  a  stand  alone  training  device. 
One  instructor  chose  B,  the  VTB  is  not  effective  as  a  training  device.  One  instructor  chose  D, 
the  physical  terrain  board  is  not  effective  as  a  training  device.  These  results  are  summarized  in 
Table  4. 

Table  4.  VTB  instructor  responses  regarding  effectiveness  as  VTB  and  physical  terrain  board  as 

stand-alone  training  devices 


Effectiveness  of  VTB  vs.  Physical  Terrain  Board 
as  Training  Device 

Instructor  Res| 

ponse 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

A.  The  VTB  is  effective  as  a  stand  alone  training  device 

X 

X 

X 

X 

X 

X 

X 

B.  The  VTB  is  not  effective  as  a  training  device 

X 

C.  The  physical  terrain  board  is  effective  as  a  stand  alone 
training  device 

X 

X 

X 

X 

D.  The  physical  terrain  board  is  not  effective  as  a  training 
device 

X 

E.  The  virtual  and  physical  terrain  boards  together  provide  the 
most  effective  training 

X 

X 

X 

X 

X 

Additional  features  or  capabilities  to  improve  the  VTB  provided  by  the  instructors  are  listed 
below: 


•  Weather  effects  -  rain/fog/clouds,  brownout/whiteout 

•  Environmental  effects  -  dust/smoke;  theater  specific  environment,  aircraft/ship  lighting 

•  Weapons  effects  -  ordnance,  lasers,  explosions, 

•  Motion  of  aircraft  -  as  if  actually  piloting  an  aircraft 

•  Fly  through  capabilities,  real  time  flight 

•  Free  placement  of  threat  vehicles 

•  Real  time  movement  of  moon  angles  and  intensity 

•  Ability  to  physically  control/move  the  moon 

•  Improve  the  shadow  modules.  The  database  did  not  generate  the  effects  one  would 
expect  to  see.  Include  a  scene  with  various  ridge  lines  and  different  moon  azimuths. 

•  Landing  Signals  Officer  (LSOs)  scenario  with  sound  and  ship  movement 
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•  Add  a  threat  module  to  replace  Visual  Threat  Recognition  and  Avoidance  Trainer 
(VTRAT) 

•  Provide  instructors  with  a  module  edit  capability. 

•  Theater  specific  enviromnent 

•  Improve  halos  to  be  more  discernable 

•  Create  a  time  lapse  demonstration 

•  Speed  scene  to  scene  transition/skip  through  module  capability 

•  Create  light  sources  to  affect  goggle  gain 

•  Always  show  halos  for  incompatible  lights 

•  One  hand  operation  on  the  controller 

•  Add  runways 

The  survey  was  concluded  by  asking  the  instructors  for  any  additional  comments.  Two 
instructors  responded  with  feedback  for  this  question.  One  instructor  commented  that  the  VTB 
is  a  "step  in  the  right  direction"  and  with  upgrades  to  modules  and  technology,  could  possibly 
replace  the  terrain  board.  The  other  instructor  noted  the  VTB’s  potential  of  being  a  stand  alone 
training  device,  its  tailorable  databases  covering  a  broad  number  of  environments,  ability  to 
provide  exposure  to  actual  NVG  flight  training  area,  portability/deployability,  threat  recognition 
and  potential  for  demonstrating  weather  effects. 

5.3.2  VTB  Student  Survey  Results 

The  total  number  of  students  responding  to  the  survey  was  76.  A  total  of  5 1  participants  were 
NVG  refresher  training  students,  31  rotary-wing  USMC  aircrew  at  Camp  Pendleton  and  20  C-17 
aircrew  at  Travis  AFB.  The  remaining  25  student  surveys  were  completed  by  USAF  personnel 
attending  the  NVG  Instructor  Course  or  the  NVG  Training  Course.  The  USMC  refresher 
students  had  an  average  of  269  NVG  flight  hours  and  the  USAF  refresher  students  had  an 
average  of  153  NVG  flight  hours.  The  refresher  student  results  considered  for  this  analysis 
included  effectiveness  ratings  for  terrain  board  learning  objectives,  overall  effectiveness  as 
compared  to  the  physical  terrain  board,  potential  improvements  to  the  VTB  and  training  points 
most  beneficial  in  familiarizing  students  with  actual  flight  operations. 

Refresher  Students: 

All  refresher  students  rated  the  training  effectiveness  of  each  objective  as  at  least  adequate  (3.0 
or  higher).  The  majority  of  the  effectiveness  ratings  were  4.0  or  greater.  USMC  refresher 
students  rated  the  effectiveness  of  each  objective  lower  than  the  USAF  students.  The  most 
notable  difference  in  rating  occurred  with  objectives  5  and  6  (related  to  demonstrations  of 
different  color/wavelengths  of  light)  and  objective  9  (demonstrate  differences  of  albedos).  The 
average  training  effectiveness  ratings  for  each  terrain  board  learning  objective  provided  by  the 
NVG  Refresher  Course  USMC  and  USAF  students  are  listed  in  Table  5. 
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Table  5.  USMC  and  USAF  NVG  refresher  course  students  average  training  effectiveness  rating 

for  each  terrain  board  learning  objective 


Terrain  Board  Learning  Objectives 

Flow  effective  the  VTB  was  in  familiarizing  you  with  the  learning 

objectives  listed 

USMC 

Refresher 

Students 

Avg  Rating  of 
Training 
Effectiveness 

USAF 
Refresher 
Students 
Average  Rating 
of  Training 
Effectiveness 

1 .  Demonstrate  both  high  and  low  contrast  terrain 

4.6 

4.8 

2.  Demonstrate  the  appearance  of  power  lines 

4.1 

4.8 

3.  Demonstrate  effects  of  illumination  level  on 
contrast/detail  over  different  terrains 

4.3 

4.9 

4.  Demonstrate  the  effect  of  no  illumination 

4.4 

4.8 

5.  Demonstrate  effects  of  different  color/wavelengths  of  light 
on  the  NVG  image 

3.4 

4.7 

6.  Demonstrate  effects  of  different  color/wavelengths  of  light 
under  different  illumination  levels  on  the  NVG  image 

3.5 

4.5 

7.  Demonstrate  effects  of  different  illumination  levels  on 
halos 

4.2 

4.8 

8.  Demonstrate  effects  of  moon  elevation  and  angle  on  the 
NVG  image 

4.4 

4.6 

9.  Demonstrate  differences  of  albedos  in  high  and  low 
contrast  areas 

3.9 

4.8 

10.  Demonstrate  effects  of  shadows  on  the  NVG  image 

4.5 

4.9 

11.  Demonstrate  the  appearance  of  cultural  features 

4.2 

4.8 

12.  Culturing  lighting  effects 

4.3 

4.7 

Refresher  students  also  rated  the  VTB's  effectiveness  at  accomplishing  the  learning  objectives 
overall  in  comparison  to  the  physical  terrain  board.  The  average  rating  that  USAF  refresher 
students  gave  was  4.9  and  the  USMC  refresher  students  average  rating  was  4.2.  The  average 
ratings  of  effectiveness  provided  by  the  USAF  NVG  Instructor  Course  and  NVG  Training 
Course  students  were  above  4.0  for  all  learning  objectives. 

Recommendations  for  future  improvements  to  the  VTB  given  were  given  by  all  students,  initial 
and  refresher,  USAF  and  USMC.  The  percentages  for  improvements  mentioned  by  all  of  the 
students  are  listed  below  for  the  top  three  mentioned  items.  All  others  were  mentioned  by  5%  of 
the  students  or  less.  They  included  the  following: 

•  Fly  through  capability  ( 1 8%) 

•  Threats:  ordnance,  lasers,  explosions,  VTRAT  integration  (13%) 

•  Theater  specific  environment/tactical  visual  features  (8%) 

•  Weather/environmental  effects  (8%) 

•  Smoother  movement  with  controller/allow  for  slower  movement 

•  Local  areas/routes 

•  Different  altitude  scenes 
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•  Edit/customization  of  modules/menu 

•  Diversify  terrain 

•  Add  cultural  features 

•  Depth  Perception 

Refresher  students  were  asked  to  rate  the  effectiveness  of  the  VTB  in  familiarizing  students  with 
assigned  flying  missions.  The  average  rating  that  USAF  refresher  students  gave  was  4.7  and  the 
USMC  refresher  students  average  rating  was  4.0.  Refresher  students  also  identified  the  VTB 
training  points  that  are  most  beneficial  in  familiarizing  students  with  actual  flight  operations. 

The  percentages  of  comments  received  for  a  particular  training  area  are  listed  for  the  top  three 
mentioned  items.  All  others  were  mentioned  by  5%  of  refresher  students  or  less. 

•  Different  moon  illuminations/positions  (31%) 

•  Terrain  and  contrast  levels  ( 1 8%) 

•  Shadowing  effects  (8%) 

•  Airfield/runway  environments 

•  Cultural  vs.  aircraft  lighting 

•  Weather 

•  Ordinance 

•  Over  water 

•  Physical  features 


6.  Conclusions 

The  results  of  the  terrain  board  learning  objectives  checklist  and  the  instructor  survey  indicate 
that  most  instructors  can  accomplish  all  terrain  board  learning  objectives  using  the  VTB. 
However,  a  few  instructors  identified  specific  learning  objectives  that  they  were  unable  to  fully 
accomplish.  The  learning  objectives  noted  as  not  being  able  to  fully  accomplish  using  the  VTB 
by  three  instructors  were:  demonstrate  effects  of  different  color  light  under  different  illumination 
levels  on  an  NVG  image,  demonstrate  effects  of  different  illumination  levels  on  halos, 
demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG  image  and  demonstrate  the  effects 
of  shadows  on  the  NVG  image.  Two  instructors  did  not  respond  with  "Yes  or  No"  but  instead 
with  comments  on  demonstrate  the  effects  of  no  illumination  objective.  Although  most 
instructors  indicated  that  the  VTB  does  accomplish  each  learning  objective,  the  presence  of  some 
instructors  choosing  "No"  or  not  responding  "Yes  or  No"  but  including  comments  indicates  that 
there  are  some  improvements  that  should  be  made  or  additional  training  devices  that  should  used 
to  support  the  VTB  for  teaching  these  learning  objectives. 

In  its  current  configuration,  external  light  sources  (including  the  moon)  and  associated  halos 
contained  in  VTB  scenes  are  simulated  and  do  not  stimulate  the  NVGs  as  accurately  as  actual 
real-world  emissive  light  sources.  The  physical  terrain  board  is  equipped  with  incandescent  light 
sources  that  vary  in  color  and  intensity,  and  are  used  to  stimulate  the  NVGs  to  produce  the 
desired  effects.  The  physical  terrain  board  lighting  system  used  for  the  moon  simulation  also 
drives  the  gain  response  of  the  NVG,  when  lights  are  in  the  field  of  view  of  the  NVG.  The 
results  of  the  learning  objectives  checklist  with  respect  to  demonstrations  of  different  color  lights 
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and  halos  indicate  that  improvements  to  the  VTB  to  allow  for  better  demonstration  of  lighting 
effects  and  halos  are  warranted.  The  differences  between  the  VTB  and  physical  terrain  board 
lighting  systems  may  also  contribute  to  the  limitations  in  the  effects  of  moon  angle  and  shadow 
demonstrations  noted  by  three  instructors. 

The  overall  ease  of  use  of  the  VTB  was  rated  by  instructors  as  above  satisfactory  and  nearly 
somewhat  easy  with  an  average  rating  of  3.8.  The  instructors  were  questioned  about  the 
usability  of  the  VTB  pertaining  to  the  video  controller,  menu  navigation,  the  classroom 
environment  and  the  teaching  modules  provided.  Over  half  of  the  instructors  commented  that 
the  ability  to  edit/customize  the  modules  and  menu  would  improve  the  usability  of  the  VTB. 

This  data  suggests  that  one  improvement  to  enhance  the  usability  of  the  VTB  in  more  than  one 
area  is  to  develop  the  ability  to  customize  the  training  modules  based  on  the  needs  of  the 
instructor.  Four  instructors  also  noted  that  a  quicker  scene -to-scene  transition  time  would 
improve  the  usability  of  the  VTB.  Also,  some  instructors  mentioned  that  a  one-handed  control 
would  improve  usability.  One  instructor  noted  difficulty  holding  NVGs  due  to  use  of  Clip-On 
Power  Source  (CLOPs)  and  manipulating  the  VTB  simultaneously.  These  comments  suggest 
that  the  instructors  were  holding  the  NVGs  in  one -hand  and  the  video  controller  gamepad  in  the 
other.  One  solution  might  be  to  wear  the  NVGs  on  a  helmet  to  free  up  both  hands  for  using  the 
gamepad.  Another  solution  might  be  the  implementation  of  a  controller  that  can  be  more  easily 
operated  with  one  hand  (i.e.,  a  remote  control  device). 

All  instructors  rated  the  training  effectiveness  of  the  12  terrain  board  learning  objectives  at  3.8  or 
above  using  the  5 -point  scale,  indicating  that  the  instructors  found  the  VTB  to  be  an  effective 
tool  for  demonstrating  most  aspects  of  the  terrain  board  learning  objectives. 

One  USN/USMC  instructor  rated  at  a  level  of  ineffective  the  demonstration  of  the  effects  of 
different  color/wavelengths  of  light  on  the  image  and  under  different  illumination  levels  on  the 
image.  The  instructor  identified  the  reason  for  these  ratings  as  a  need  for  improved 
color/wavelength  for  aided  and  unaided  for  the  VTB.  The  average  rating  given  for  these  two 
objectives  were  4.0  (on  NVG  image)  and  3.7  (on  NVG  image  at  different  illumination  levels), 
indicating  that  the  majority  of  the  instructors  could  accomplish  most  of  the  teaching  points 
associated  with  the  two  objectives.  These  results  are  consistent  with  the  terrain  board  objectives 
checklist  findings  discussed  previously. 

The  USMC  refresher  students  also  rated  objectives  related  to  the  demonstration  of  different 
colors/wavelengths  of  light  notably  lower  than  the  USAF  students.  The  instructors  that 
commented  on  this  learning  objective  were  all  USMC  instructors.  This  indicates  that  the 
learning  objective  related  to  the  demonstration  of  colors/wavelengths  of  light  may  be  a  more 
crucial  learning  objective  to  the  USMC. 

The  USMC  refresher  students  also  rated  consistently  lower  in  for  effectiveness  of  all  objectives 
and  over  all  effectiveness  when  compared  to  the  physical  terrain  board.  The  different  flight 
missions  of  the  USMC  and  USAF  and  the  finding  that  USMC  refresher  students,  on  average,  had 
nearly  twice  as  many  NVG  flight  hours  may  contribute  to  this  difference.  The  USMC  refresher 
students  may  be  more  discerning  observers  given  the  nature  of  the  flight  missions  they  are 
assigned  (low-level).  Also,  many  effectiveness  ratings  for  the  USMC  students  were  marked  as 
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"not-applicable"  indicating  that  they  may  have  had  little  or  no  exposure  to  the  particular 
objective  indicated  on  the  survey. 

Other  potential  demonstrations  identified  by  the  instructors  for  effective  VTB  training  included 
environmental  effects.  Environmental  effects  are  not  a  current  capability  of  the  VTB  and  were 
rated  very  low  on  the  learning  objective  scales  because  they  are  not  a  feature  of  the  version  of  the 
VTB  (V2.0c)  evaluated  for  this  effort.  Over  half  of  the  respondents  listed  environmental  effects 
as  an  effectively  trainable  learning  objective.  Also,  enviromnental/weather  effects  were 
commented  on  repeatedly  in  the  survey  by  the  instructors.  This  is  an  indication  that  the  inclusion 
of  environmental/weather  effects  will  be  an  important  feature  to  include  during  further 
development  of  the  VTB. 

The  instructor  feedback  regarding  learning  objectives  that  were  not  possible  to  accomplish  using 
the  physical  terrain  board  that  can  be  demonstrated  on  the  VTB  showed  various  responses.  The 
only  response  that  was  given  by  two  different  instructors  was  the  weather,  dust  and  smoke 
effects  as  a  learning  objective.  As  stated  previously,  this  answer  arose  in  several  different 
questions.  Other  objectives  listed  included  weapons  employment,  mission  planning  and 
rehearsal  and  demonstration  of  various  threats. 

The  instructor  feedback  regarding  learning  objectives  not  possible  to  accomplish  on  the  VTB 
that  can  be  demonstrated  on  the  physical  terrain  board  also  varied  greatly.  One  objective  that 
was  mentioned  consistently  was  effects  related  to  moon  angle/position.  Six  of  the  eleven 
instructors  noted  that  the  demonstration  of  the  effects  of  moon  elevation  angle  and  position  could 
be  more  readily  demonstrated  on  the  physical  terrain  board.  This  shows  that  this  is  an  important 
capability  used  whose  absence  is  noticed  among  VTB  users.  Other  capabilities  of  the  physical 
terrain  board  that  were  noted  as  being  not  capable  on  the  VTB  included  aircraft/ship  lighting 
effects,  limitation  of  pre-loaded  modules;  and  goggle  gain  effects  while  viewing  incompatible 
light  source. 

Instructors  were  asked  to  mark  each  statement  that  applied  in  a  comparison  between  the  physical 
terrain  board  and  the  VTB.  Two  statements  each  were  given  for  the  independent  use  of  the  two 
systems:  (A)  the  VTB  is  effective  as  a  stand  alone  training  device  or  (C)  the  physical  terrain 
board  is  effective  as  a  stand  alone  training  device.  Two  stated  the  opposite:  (B)  the  VTB  is  not 
effective  as  a  stand-alone  training  device  and  (D)  the  physical  terrain  board  is  not  effective  as  a 
stand-alone  training  device.  One  more  statement  was  added  indicating  that  (E)  the  virtual  and 
physical  terrain  boards  together  provide  the  most  effective  training. 

The  results  indicate  that  7  instructors  surveyed  believe  that  the  VTB  is  effective  as  a  stand  alone 
training  device.  Five  instructors  indicate  that  the  most  effective  training  method  is  using  these 
two  devices  together.  The  selection  of  either  A  or  E  by  all  of  the  instructors  indicates  that  they 
consider  the  VTB,  as  either  one  element  for  training  with  the  physical  terrain  board  or  by  its  sole 
use  as  a  trainer,  beneficial  to  the  trainee  for  effectively  conveying  the  terrain  board  learning 
objectives. 

VTB  instructors  identified  improvements  that  they  would  recommend  for  the  VTB.  The  list 
given  by  the  instructors  varied  greatly  showing  that  there  is  still  room  for  improvement  for  the 
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VTB.  Again,  environmental/weather  effects  and  real  time  movement  of  moon  angles  and 
intensity  was  noted  in  this  section. 

Additional  comments  were  requested  at  the  completion  of  the  survey.  Two  instructors  responded 
to  this  section.  Both  comments  given  indicated  that  these  individuals  see  the  VTB  as  a 
progression  in  technology.  At  the  same  time,  these  instructors  indicate  that  although  is  a  step 
forward,  there  are  still  ways  that  it  can  be  improved. 

Overall  the  survey  results  indicated  that  the  majority  of  the  instructors  surveyed  believe  that  the 
VTB  adequately  and  effectively  trains  for  the  current  terrain  board  learning  objectives.  Most  of 
ratings  given  by  instructors  for  effectiveness  ranged  between  3,  adequate  and  5,  very  effective. 

As  the  individual  objectives  are  examined,  however,  improvements  still  can  be  made  in  different 
learning  areas  to  make  VTB  more  effective. 

The  most  prominently  mentioned  capabilities  for  VTB  development  are  environmental/weather 
effects  and  the  ability  to  change  moon  angle  and  position.  The  addition  of  these  capabilities 
could  potentially  eliminate  several  of  the  items  listed  as  limitations  of  the  VTB.  Regarding  the 
usability  of  the  VTB,  the  ability  to  modify  and  edit  the  training  modules  was  mentioned 
frequently. 

Overall,  the  instructors  rated  the  VTB  favorably  as  part  of  their  training  programs  as  either  a 
stand  alone  training  device  or  when  used  in  conjunction  with  the  physical  terrain  board.  In  the 
future,  the  suggestions  given  for  additional  learning  objectives  can  be  used  to  develop  the  VTB 
further  for  relevant  training  in  other  areas  in  addition  than  those  currently  trained  with  the  terrain 
board. 

Both  NVG  Refresher  and  NVG  Instructor  Course  students  listed  areas  of  improvement  for  the 
VTB.  Many  echoed  the  responses  of  the  instructors  with  respect  to  including  fly-through 
capability,  threats,  weather/environmental  effects,  different  altitude  scenes,  and 
edit/customization  of  modules/menu.  The  instructors  rated  lower  than  all  of  the  refresher 
students  in  only  two  areas:  effects  of  moon  elevation  and  angle  on  the  NVG  image  and  cultural 
lighting  effects.  One  item  mentioned  frequently  by  USAF  students  was  the  potential  use  of  the 
VTB  as  a  platform  for  the  VTRAT. 

Across  all  ratings;  the  effectiveness  of  VTB  at  accomplishing  specific  learning  objectives,  at 
familiarizing  students  with  assigned  flying  missions  and  overall  effectiveness,  the  USMC 
students  rated  slightly  lower  than  the  USAF  students.  Possible  explanations  for  this  are  that  the 
USMC  refresher  students  have  more  NVG  flight  hours  on  average  than  the  USAF  refresher 
students  (269:153)  and  were  more  sensitive  to  the  limitations  of  the  VTB  and  perhaps  more 
discerning  in  their  evaluation  of  learning  objectives.  Also,  the  USMC  and  USAF  refresher 
students’  flying  missions  are  different,  and  thus  would  likely  offer  different  perspectives.  USMC 
students  consisted  of  rotary-wing  aircrew,  whereas  USAF  students  were  fixed-wing  pilots. 

Refresher  students  commented  on  VTB  training  points  that  are  most  beneficial  in  familiarizing 
students  with  actual  flight  operations.  Their  responses  reflected  current  and  potential  training 
points  that  could  be  shown  on  the  VTB.  The  USMC  students  mentioned  weather,  ordnance,  over 
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water  flight,  and  shadowing.  The  USAF  students  noted  halo  effects,  cultural  vs.  aircraft  lighting 
and  shadowing.  As  listed  previously,  all  refresher  students  noted  varying  illumination, 
terrain/contrast  and  fly  through  capability. 

Nonetheless,  these  results  suggest  that  different  flying  communities  will  have  different  training 
requirements  with  respect  to  VTB  learning  objectives.  This  lends  further  support  to  the  concept 
of  providing  a  better  capability  to  customize/edit  training  modules  to  accommodate  these 
differences  in  requirements. 

7.  Recommendations 

Based  on  the  instructor  and  student  feedback  obtained  for  this  evaluation,  the  VTB  does 
effectively  accomplish  most  aspects  of  the  established  terrain  board  learning  objectives. 

However,  further  development  is  recommended  in  order  to  increase  the  VTB’s  training 
effectiveness  for  specific  objectives  and  to  more  fully  realize  its  training  value. 

It  is  recommended  that  future  development  efforts  of  the  VTB  be  aimed  at  improving  system 
usability  and  the  realism  of  NVG  effects  in  the  specific  areas  outlined  below.  These 
recommendations  are  based  upon  the  comments/ratings  obtained  from  both  qualified  NVG 
instructors  and  NVG-experienced  aircrew  and  are  designed  to  enhance  the  training  value  of  the 
VTB.  The  following  list  provides  areas  in  which  development  efforts  should  be  focused  in 
future  updates  to  the  VTB. 

Weather  and  environmental  effects.  Recommend  effects  such  as  snow,  rain,  fog,  dust,  smoke, 
etc.  be  added  to  future  training  modules  of  the  VTB.  Provided  that  the  VTB  technology  has  the 
capability,  the  majority  of  instructors  noted  that  the  addition  of  weather  and  environmental 
effects  would  relieve  some  of  the  limitations  of  the  VTB  in  its  current  state. 

Add  threat  module(s).  Recommend  adding  threat  modules  to  simulate  threat  effects  to  include 
ordnance,  lasers,  and  explosions.  The  addition  of  a  threat  module  would  expose  students  to 
NVG  effects  caused  by  threats  and  could  possibly  replace  or  be  integrated  with  VTRAT. 

Lunar  azimuth  and  elevation  control.  A  more  intuitive  ability  and  expanded  capability  to  change 
lunar  phase,  elevation,  and  azimuth.  Adding  this  capability  may  address  the  some  limitations 
encountered  in  the  VTB  shadow  module. 

Within  the  existing  configuration,  this  recommended  improvement  could  be  accomplished  by 
displaying  scenes  in  a  logical,  timed  sequence  which  gives  the  effect  of  moon  position  changes. 
This  could  be  done  by  using  the  current  configuration  and  runtime  software  to  develop  additional 
instructor  modules  that  illustrate  the  effects  of  moon  elevation,  azimuth  and  phase.  For  example, 
a  module  with  stepped  changes  in  elevation  over  several  different  scenes  can  be  created  to 
demonstrate  the  impact  that  these  changes  have  upon  the  scene.  The  instructor  could  emphasize 
the  changing  shadows  and  other  effects  as  they  relate  to  elevation  changes. 

An  alternate  and  preferred  method  to  control  the  moon  phase,  elevation  and  azimuth  is  to 
develop  an  on-the-fly  ability  to  manipulate  the  moon  position  as  desired  by  the  instructor.  Moon 
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control  will  affect  the  shadows  and  illumination  displayed  in  the  scenes  and  can  be  changed  in 
real  time  at  the  discretion  of  the  instructor.  In  this  case,  the  control  parameters  would  be  similar 
to  those  demonstrated  on  the  physical  terrain  board. 

Customization  of  modules.  VTB  software  should  allow  the  instructor  to  develop  custom 
modules  tailored  to  each  unit's  training  needs.  For  example,  instead  of  having  to  show  an  entire 
module  as  it  is  sequenced  by  the  software  designer,  the  instructor  would  be  able  to  select  the 
scenes  that  they  wish  to  use  for  their  class  by  designing  a  custom  module  to  demonstrate  to  their 
students. 

Expansion  and  improvement  of  the  shadow  module.  Create  an  expanded  selection  of  scenes  in 
shadow  module  and  improve  upon  the  existing  scenes.  The  database  should  be  improved  to 
generate  shadows  and  objects  in  shadows  as  they  are  expected  to  be  seen.  The  shadows  module 
should  also  be  expanded  to  include  more  complex  scenes.  For  example,  a  scene  with  shadows 
created  by  various  ridge  lines  and/or  objects  in  the  shadows. 

Improve  halos.  The  halo  representations  should  be  improved  to  provide  an  accurate 
representation  of  how  halos  look  through  NVGs.  One  issue  is  that  halo  size  decreases  or  remains 
constant  when  zooming  in  towards  a  light  in  the  demonstration.  Also,  in  certain  scenes  halos 
from  lights  are  either  too  opaque  or  too  bright. 

Improve  different  color/wavelength  of  light  depiction.  Improvements  to  color  light  depiction  for 
unaided  view  are  recommended.  Colored  lights  within  the  VTB  scene  are  difficult  to  detect 
unaided.  A  light  bar  can  be  used  as  an  auxiliary  training  device  for  demonstrations  of  colored 
lights  and  how  they  look  aided  and  unaided.  The  use  of  a  light  bar  may  provide  a  better  a 
physics-based  demonstration  of  how  NVGs  react  to  multi-colored  and/or  incompatible  lighting. 

Improve  goggle  gain  effects:  Creating  incompatible  lights  that  can  affect  goggle  gain  in  the 
NVG  is  recommended.  Incompatible  lights,  such  as  the  moon  ,  do  not  gain  down  the  goggle 
accurately.  The  incorporation  of  the  light  bar  mentioned  above  might  also  serve  to  demonstrate 
NVG  gain  effects. 

Zero  illumination  scene(s).  Currently,  none  of  the  modules  available  depict  a  scene  with  no 
illumination.  As  part  of  the  instruction,  it  is  recommend  turning  out  all  room  lights  in  a  light¬ 
tight  dark  room  to  demonstrate  that  at  least  some  light  is  needed  for  the  NVG  to  amplify  and 
provide  the  user  a  perceptible  NVG  image. 

Improve  controller.  Recommend  exploration  of  alternative  control  options  to  replace  the 
existing  video  controller  in  order  to  allow  one-handed  control  of  the  VTB. 

The  strong  presence  of  each  of  these  items  in  the  comments  sections  of  the  survey  indicates  that 
there  is  a  need  for  improvement  and/or  additional  capabilities  in  these  areas.  By  further 
development  in  the  recommended  areas,  the  VTB's  potential  for  more  effective  training  of 
current  and  new  terrain  board  learning  objectives  should  be  increased. 
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8.  Summary 


Night  Readiness,  the  VTB  manufacturer,  plans  to  expand  several  capabilities  of  the  VTB  in  the 
near  future.  The  version  evaluated  here  was  2.0c.  Current  plans  for  future  VTB  development 
include  weather/environmental  effects,  improved  database  generation  using  commercially 
available  software,  advanced  real-world  scenes  documentation  and  fly  through  capabilities  using 
government-off-the-shelf  (GOTS)  software.  The  addition  of  these  capabilities  will  allow  for: 

•  Mission  pre-brief  and  preview 

•  Post-mission  critique 

•  Mishap  Investigation 

•  Geospecific  databases 

•  Rapid  development  of  1000  square  mile  databases 

•  Demonstration  of  weather/environmental  effects 

Future  development  in  the  areas  listed  will  address  most  of  the  improvements  desired  by  the 
instructors  and  students  surveyed.  The  results  of  the  evaluation  conducted  show  that  the  VTB 
effectively  accomplishes  most  of  the  learning  objectives  of  the  terrain  board  and  with  the 
implementation  of  auxiliary  devices  (e.g.,  light  bar)  and  improvements  to  database  scene  realism, 
may  accomplish  all  of  them  effectively.  The  development  plans  for  the  VTB  currently  underway 
along  with  the  improvements  recommended  in  this  evaluation  will  provide  the  most  balanced 
approach  to  improving  the  usability,  versatility  and  training  effectiveness  of  the  VTB. 


32 


9.  References 


Antonio,  J.  C.,  Berkley,  W.  E.,  Fiedler,  G.,  &  Joralmon,  D.  Q.  (2004).  Night  vision  goggle 
training  course,  version  3(1  ed.).  Mesa  AZ:  Air  Force  Research  Laboratory,  Human 
Effectiveness  Directorate,  Warfighter  Readiness  Research  Division. 

Antonio,  J.  C.,  Berkley,  W.  E.,  &  Joralmon,  D.  Q.  (1994).  Night  vision  goggle  training  course  (1 
ed.).  Williams  Air  Force  Base,  AZ:  Armstrong  Laboratory,  Aircrew  Training  Research 
Division. 

Antonio,  J.  C.,  Berkley,  W.  E.,  &  Joralmon,  D.  Q.  (1998).  Night  vision  goggle  training  course, 
version  2(1  ed.).  Mesa  AZ:  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate,  Warfighter  Readiness  Research  Division. 

Joralmon,  D.Q.  (2004).  Effects  of  telecommunications  support  on  the  development  and 

administration  of  a  military  training  course.  Unpublished  doctoral  dissertation,  Arizona 
State  University,  Tempe. 

Joralmon,  D.  Q.,  &  Antonio,  J.  C.  (1992).  Night  vision  goggle  instructional  series  [Videotape]. 
Mesa,  AZ:  University  of  Dayton  Research  Institute. 

Martin,  E.  L.,  Berkley,  W.  E.,  Riegler,  J.  T.,  Good,  J.  N.,  Anderson,  G.  M.,  Tooruella,  J.  R., 
Joralmon,  D.  Q.,  House,  D.  A.,  Andrews,  D.  H.,  Griffith,  J.  S.,  &  Fuller,  J.  H.  (2004). 
Night  vision  goggle  training  in  SUPT  and  IFF:  A  feasibility  study  (AFRL-HE-  AZ-TR- 
2004-0199).  Mesa  AZ:  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate, 
Warfighter  Readiness  Research  Division. 


33 


10.  Acronyms 


AFB 

Air  Force  Base 

CDRL 

Contract  Data  Requirements  List 

COTS 

commercial  off-the-shelf 

DCS 

DMO  Control  Station 

DVT 

Desktop  Visualization  Trainer 

HMD 

Helmet  Mounted  Display 

HUD 

Heads-Up  Display 

l/ITSEC 

Interservice/Industry  Training,  Simulation  &  Education  Conference 

IG 

Image  Generator 

LANTIRN 

Low-Altitude  Navigation  and  Targeting  Infrared  for  Night 

MRS 

Mesa  Research  Site 

NVG 

Night  Vision  Goggle 

NVTS 

Night  Vision  Training  System 

OTW 

Out-the  Window 

PETS 

Performance  Evaluation  and  Tracking  System 

PFPS 

Portable  Flight  Planning  Software 

PI 

Principal  Investigator 

PNVG 

Panoramic  Night  Vision  Goggle 

RCG 

Rickard  Consulting  Group 

RSC 

Renaissance  Sciences  Corporation 

SBIR 

Small  Business  Innovative  Research 

SME 

Subject  Matter  Expert 

SOW 

Statement  of  Work 

SPINS 

Special  Instructions 

SSOW 

Subcontractor  Statement  of  Work 

TMWP 

Technical  Management  Work  Plan 

UWB 

Ultra  Wide  Band 

ViPRS 

Video  Processor  for  Real-time  Simulation 

VTB 

Virtual  Terrain  Board 
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1 1 .  Appendix  A:  VTB  Instructor  Survey 


Virtual  Terrain  Board  Survey 
Instructors 


Name:  _ 
Rank:  _ 
Service: 


Courses  Administered: 

Aircraft: _ 

Date: 


1.  Are  you  currently  qualified  as  a  Terrain  Board  instructor?  Yes _  No _ 

2.  Physical  Terrain  Board  experience  (hrs  as  an  instructor): 

o  0 
o  1-100 
o  101-200 
o  >  200 

3.  Virtual  Terrain  Board  experience  (hrs  as  an  instructor): 

o  0  hrs 
o  1  to  3  hrs 
o  4  to  8  hrs 
o  More  than  8  hrs 

4.  Please  rate  the  ease  of  use  of  the  overall  VTB  system  (from  an  instructor’s  point  of  view): 

1-  Very  Difficult  -  much  improvement  required 

2-  Somewhat  Difficult  -  certain  features  need  improvement 

3-  Satisfactory  -  ease  of  use  is  acceptable  but  could  be  improved 

4-  Somewhat  Easy  -  Minor  improvements  would  help  but  not  required 

5-  Very  Easy  -  No  improvements  required 

5.  How  would  you  improve  the  usability  of  the  VTB  with  respect  to?: 

a.  Video  controller 


b.  Menu  navigation 
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c.  Classroom  environment 


d.  Teaching  modules  provided 


6.  Please  use  the  rating  scale  below  to  rate  the  effectiveness  of  the  VTB  in  accomplishing  the  terrain 
board  learning  objectives: 

1-  Very  ineffective  -  unable  to  accomplish  any  learning  objectives 

2-  Ineffective-  unable  to  accomplish  most  learning  objectives 

3-  Adequate  -  able  to  accomplish  some  learning  objectives 

4-  Effective  -  able  to  accomplish  most  learning  objectives 

5-  Very  effective  -  able  to  accomplish  all  learning  objectives 

NA-  Not  Applicable  -  this  objective  is  not  part  of  our  training  curriculum 
UN-  Unsure-  have  not  had  enough  experience  with  this  parameter 

a.  Demonstrate  both  high  contrast  and  low  contrast  terrain  _ 

b.  Demonstrate  effects  of  illumination  level  on  contrast/detail  over  different  terrains _ 

c.  Demonstrate  the  appearance  of  cultural  features  (i.e.,  roads,  bridges) _ 

d.  Cultural  lighting  (i.e.,  halo  effects,  reflections) _ 

e.  Demonstrate  the  effects  of  environmental  factors  (i.e.,  weather,  dust,  smoke) _ 

f.  Demonstrate  differences  in  albedo  for  different  objects _ 

g.  Demonstrate  the  appearance  of  power  lines _ 

h.  Demonstrate  the  effect  of  no  illumination _ 

i.  Demonstrate  effects  of  different  color/wavelengths  of  light  on  the  NVG  image _ 

j.  Demonstrate  effects  of  different  color/wavelengths  of  light  under  different  illumination  levels 

on  the  NVG  image _ 

k.  Demonstrate  effects  of  different  illumination  levels  on  halos _ 

l.  Demonstrate  effects  of  shadows  on  the  NVG  image _ 

m.  Demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG  image _ 

7.  Please  rate  the  extent  to  which  the  VTB  allows  you  to  accomplish  the  overall  learning  objectives  as 

compared  to  the  physical  terrain  board.  _ 
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8.  Please  explain  the  limitations  of  those  learning  objectives  you  marked  ineffective  or  very  ineffective. 


9.  Please  list  any  other  NVG  training  objectives  (not  listed  above)  that  could  be  effectively  trained  using 
the  VTB. 


10.  Please  list  any  learning  objectives  that  were  not  possible  to  accomplish  on  the  physical  terrain  board 
that  can  be  demonstrated  on  the  VTB. 


1 1 .  Please  list  any  learning  objectives  that  were  not  possible  to  accomplish  on  the  VTB  that  can  be 
demonstrated  on  the  physical  terrain  board. 


12.  Regarding  the  physical  terrain  board  and  VTB:  (circle  all  that  apply) 

A.  The  virtual  terrain  board  is  effective  as  a  stand  alone  training  device 

B.  The  virtual  terrain  board  is  not  effective  as  a  training  device 

C.  The  physical  terrain  board  is  effective  as  a  stand  alone  training  device 

D.  The  physical  terrain  board  is  not  effective  as  a  training  device 

E.  The  virtual  and  physical  terrain  boards  together  provide  the  most  effective  training 

13.  What  improvements  would  you  recommend  for  the  VTB? 


Additional  Comments: 
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12.  Appendix  B:  Terrain  Board  Learning  Objectives 
Checklist 

Terrain  Board  Learning  Objectives  Checklist 

Name:  _  Unit: _ 

Please  use  the  checklist  below  to  indicate  whether  the  Virtual  Terrain  Board  allows  you  to  accomplish 
each  specific  learning  objective.  If  necessary,  please  use  the  comment  section  to  elaborate  on  a  specific 
objective.  Select  “YES”  if  you  are  able  to  accomplish  the  learning  objective  on  the  VTB.  Select  “NO”,  if 
you  are  unable  to  accomplish  the  objective. 


Learning  Objectives 

YES 

NO 

1 .  Demonstrate  differences  in  albedo  for  different  objects 

2.  Demonstrate  the  appearance  of  power  lines 

3.  Demonstrate  effects  of  low  illumination  on  contrast/detail 
over  different  terrains 

4.  Demonstrate  the  effect  of  no  illumination 

5.  Demonstrate  effects  of  different  color/wavelengths  of  light 
on  the  NVG  image 

6.  Demonstrate  effects  of  different  color/wavelengths  of  light 
under  different  illumination  levels  on  the  NVG  image 

7.  Demonstrate  effects  of  different  illumination  levels  on  halos 

8.  Demonstrate  effects  of  moon  elevation  and  angle  on  the 

NVG  image 

9.  Demonstrate  differences  of  albedos  in  high  and  low 
contrast  areas 

10.  Demonstrate  effects  of  shadows  on  the  NVG  image 

1 1 .  Demonstrate  the  effects  of  various  illumination  levels 
on  image  quality 

Comments: 

Objective  # 

Objective  # 
Objective  # 
Objective  # 
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Objective  # 


13.  Appendix  C:  VTB  Students  Survey 


Virtual  Terrain  Board  Survey 
Students 


Name:  _ 
Rank:  _ 
Service: 


Date: _ 

Course  Attended: 
Aircraft: 


Total  Flight  Flours: _  Total  NVG  Flight  Flours: _  Unit: _ 

NVG  Related  Flight  Qualifications  (e.g.,  NSI,  etc.): _ 

Past  NITE  Lab  NVG  Training  Courses  completed: _ 

1 .  Please  use  the  rating  scale  below  to  rate  how  effective  the  VTB  was  in  familiarizing  you  with  the 
learning  objectives  listed: 

1-  Very  ineffective  -  unable  to  accomplish  any  learning  objectives 

2-  Ineffective-  unable  to  accomplish  most  learning  objectives 

3-  Adequate  -  able  to  accomplish  some  learning  objectives 

4-  Effective  -  able  to  accomplish  most  learning  objectives 

5-  Very  effective  -  able  to  accomplish  all  learning  objectives 

NA-  Not  Applicable  -  this  objective  is  not  part  of  our  training  curriculum 
UN-  Unsure-  have  not  had  enough  experience  with  this  parameter 


a.  Demonstrate  both  high  contrast  and  low  contrast  terrain  _ 

b.  Demonstrate  effects  of  illumination  level  on  contrast/detail  over  different  terrains _ 

c.  Demonstrate  the  appearance  of  cultural  features  (i.e.,  roads,  bridges) _ 

d.  Cultural  lighting  (i.e.,  halo  effects,  reflections) _ 

e.  Demonstrate  the  effects  of  environmental  factors  (i.e.,  weather,  dust,  smoke) _ 

f.  Demonstrate  differences  in  albedo  for  different  objects _ 

g.  Demonstrate  the  appearance  of  power  lines _ 

h.  Demonstrate  the  effect  of  no  illumination _ 

i.  Demonstrate  effects  of  different  color/wavelengths  of  light  on  the  NVG  image _ 

j.  Demonstrate  effects  of  different  color/wavelengths  of  light  under  different  illumination  levels 

on  the  NVG  image _ 

k.  Demonstrate  effects  of  different  illumination  levels  on  halos _ 

l.  Demonstrate  effects  of  shadows  on  the  NVG  image _ 

m.  Demonstrate  effects  of  moon  elevation  and  angle  on  the  NVG  image _ 


2.  Please  rate  the  extent  to  which  the  VTB  allows  you  to  accomplish  the  overall  learning  objectives  as 
compared  to  the  physical  terrain  board.  _ 
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3.  Please  explain  the  limitations  of  those  you  marked  ineffective  or  very  ineffective. 


4.  Please  list  any  other  NVG  training  objectives  (not  listed  above)  that  could  be  effectively  trained  using 
the  VTB. 


5.  What  improvements  would  you  recommend  for  VTB? 


6.  Regarding  the  physical  terrain  board  and  VTB:  (check  all  that  apply) 

F.  The  virtual  terrain  board  is  effective  as  a  stand  alone  training  device 

G.  The  virtual  terrain  board  is  not  effective  as  a  training  device 

H.  The  physical  terrain  board  is  effective  as  a  stand  alone  training  device 

I.  The  physical  terrain  board  is  not  effective  as  a  training  device 

J.  The  virtual  and  physical  terrain  boards  together  provide  the  most  effective  training 

K.  I  have  only  been  exposed  to  the  VTB. 


7.  Please  list  any  learning  objectives  that  were  not  possible  to  accomplish  on  the  physical  terrain  board 
that  can  be  demonstrated  on  the  VTB. 


8.  Please  list  any  learning  objectives  that  were  not  possible  to  accomplish  on  the  VTB  that  can  be 
demonstrated  on  the  physical  terrain  board. 
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For  refresher  students  only: 


1-  Very  ineffective  -  unable  to  accomplish  any  learning  objectives 

2-  Ineffective-  unable  to  accomplish  most  learning  objectives 

3-  Adequate  -  able  to  accomplish  some  learning  objectives 

4-  Effective  -  able  to  accomplish  most  learning  objectives 

5-  Very  effective  -  able  to  accomplish  all  learning  objectives 

NA-  Not  Applicable  -  this  objective  is  not  part  of  our  training  curriculum 
UN-  Unsure-  have  not  had  enough  experience  with  this  parameter 

9.  Please  use  the  above  scale  to  rate  the  effectiveness  of  the  VTB  in  familiarizing  students  with  assigned 
flying  missions. _ 


10.  What  specific  VTB  training  points  are  most  beneficial  in  familiarizing  students  with  actual  flight 
operations? 
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